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A Oe INTRODUCTION 

Over the last several years, the Department of Defense 
(DoD) has sought to improve the personnel security screening 
process. This desire was prompted primarily by the rash of 
security breaches over the last several years. One area where 
improvement is desired is the area of information assessment. 
This assessment occurs when personnel at adjudication 
facilities evaluate Personnel Security Investigation (PST) 
documentation obtained from the Defense Investigative Services 
(DIS). 

The Report of Investigation (ROI) is the primary 
information COndiee between these agencies. More 
specifically, the ROI is the written interface between the DIS 
Field Agent who conducted the PSI and the adjudicator 
responsible for assessing the case. 

One objective of all involved in the security screening 
process is to improve the quality and clarity of the ROI. ROI 
improvements will hopefully increase reader comprehension and 
general document effectiveness. Several important factors 
that affect the quality of the ROI are the document design, 
organization, and writing style. 

Two studies (Suchan 1989, and 1992) have examined these 
factors for the Defense Personnel Security Research and 


Education Center (PERSEREC). Suchan’s research is outlined in 


the PERSEREC technical report titled "Comparison of High and 
Low-impact Report of Investigations on Adjudicators’ Decision 
Making: A Pilot Study". His primary focus in that study is to 
determine the affect that organization, document design, and 
Stylistic factors have on the decision making ability of 
adjudicators. This thesis builds on that work. 1G 
specifically examines how the ROI document design, 
organization and writing style affect the following: 


@® The impressions of the adjudicators concerning the case 
information; 


® The perception of the adjudicators about the field agent 
who wrote the case; 


® The preference of the adjudicators for one of the two case 
styles used in the study; 


® The case processing time required by the adjudicators. 


The remainder of this chapter will discuss in more detail 
the background of Suchan’s research, present the specific 
research question that the thesis will attempt to answer, and 


then end with a preview of the remaining chapters. 


A. BACKGROUND 

As stated earlier, this thesis continues the ongoing 
research of Suchan. His research iS concerned with the 
adjudicators ability to accurately read ROI’s and make 
reliable and consistent judgements about a subject’s security 


worthiness. Suchan’s research objective is to determine if 


changes in document design, organization, and writing style, 
will enable adjudicators to make higher quality decisions. 
jis decision making ability of the adjudicator is the crux of 
the personnel security process, which has the purpose of 
weeding-out subjects who are capable of committing espionage. 

Drawing from the existing body of literature that defines 
the qualities of a clearly written document, Suchan composited 
those qualities into two ROI’s. The clearly written ROI’s 
contained organizational features such as internal previews 
and clear transitional tags; document design characteristics 
Such as strategic use of lists, bullets, heading and white- 
Space; and stylistic traits such as active verbs and right- 
branching modifiers. The total of these features is commonly 
referred to as a "high-impact" style. 

The converse of the high-impact style is referred to as 
the "low-impact" style. This is the status-quo style field 
agents currently use when writing ROI’s. This style includes 
features such as poor document design, "wall-to-wall" prose, 
no headings or lists, and no use of bold print or other 
highlighting devices; lack of internal previews at the 
beginning of long sections; long convoluted sentences that 
Strain short-term memory; passive verbs; and finally lack of 
old information/new information patterns to create coherence. 

Suchan tested both the high-impact and low-impact ROI’s on 
a sample of (40) adjudicators from four adjudication sites; 


(12) from the Army Central Adjudication Facility; (12) the 


Navy Central Adjudication Facility; (4) °f&rom the Arr erome— 
Security Clearance Office; and (12) from the Directorate for 
Industrial Security Clearance Review at Columbus, Ohio. His 
objective was to determine whether differences in ROI design, 
organization, and writing style affected the decision making 
ability of the tested adjudicators. 

A four part questionnaire was used to collect the 
necessary data for the research, although Suchan only analyzed 
the first part of that questionnaire. The remaining three 
parts of the questionnaire collected data concerning the 
Following: 


® The impressions of the adjudicators concerning the case 
INnfOLMatkaoen; 


® The perception of the adjudicators about the field agent 
who wrote the case; 


® The preference of the adjudicators for one of the two case 
styles used in the study; 


Suchan’s research results provide the basis for this 
thesis. The unexamined data from the remaining three parts of 
the questionnaire were used as the primary source of data for 
the analysis. The results of his study are outlined in the 
next chapter, the Literature Review, while a more detailed 
description of his research design is included in the Research 


Methodology Chapter. 


B. RESEARCH QUESTIONS 

This thesis will answer the following primary and 
secondary research question: 

1. Primary Research Questions 

This thesis’s primary objective is to analyze the 

remaining three parts oof the Suchan questionnaire. 
Specifically, it will determine if there is a statistical 
Significance in the questionnaire responses for ROI’s written 
in a high-impact style versus a low-impact style. The 
primary research questions for the thesis are as follows: 

@® Will there be statistically significant differences 
between adjudicators’ impressions of their ability to 
process case information presented in a high-impact ROI 
versus a low-impact one? 

@® Will there be statistically significant differences 
between adjudicators’ perceptions or feeling towards the 
field agents who wrote a high-impact ROI as compared to 
those who wrote the low-impact ROI? 

® Will there be statistically significant differences in 
adjudicators’ preferences for the case information 
presented in a high-impact versus the low-impact style? 

@® Will there be statistically significant differences in the 
adjudicators’ processing time of the case information 
presented in a high-impact style versus the low-impact 
style? 

2. Secondary Research Questions 
The thesis will answered the following secondary 
research questions: 

@® Will there be any statistical differences in the primary 
research questions presented above, when results from the 


adjudicators at the various central adjudication sites are 
analyzed independently? 


@ Will there be any statistical differences in the primary 
research questions presented above, when the population of 
adjudicators is broken down demographically? 

C. CHAPTER PREVIEW 
This section discusses the organization of the remainder 
of the thesis and previews each coming chapter. 
1. Literature Review 
An outline of the results of prior related research 
and a discussion of the significant published literature on 
the subject of document design, organization, and writing 
style will be presented. 
2. Research Methodology 
A detailed description of the research methodology 
used in the thesis is presented. Both case selection and 
questionnaire content are examples of topics discussed in this 
chapter. 
3. Presentation and Analysis of Adjudication Times and 
the Adjudicator’s Impressions of Case Information 
A presentation and analysis of the adjudication times 
and Part II of the questionnaire is the focus of the chapter. 
Part II covers the impressions of the adjudicators concerning 
the case information. 
4. Presentation and Analysis of Questionnaire Part III 
Results: the Adjudicator’s Perceptions of Field Agents 
Similar to the Preceding chapter, an analysis of Part 


Iit of the questionnaire and discussion of the statistical 


testing results is presented. Part III tests the perception 
of the adjudicators about the field agents who wrote the case. 
5. Presentation and Analysis of Questionnaire Part IV 
Results: the Writing Style Preferences of Adjudicators 
This chapter present the result of Part III of the 
Questionnaire. Part a og determine the adjudicators 
preferences between the high and low-impact writing style. 
6. Conclusions and Recommendations 
This chapter summarizes the conclusions and 


recommendations of the thesis. 


a ee LITERATURE REVIEW 


A. CHAPTER INTRODUCTION 

This chapter focuses on the significant published 
literature concerning the subject of document design, 
organization, and writing style. Much of the document 
improvement strategies discussed in this literature was 
incorporated into the revised high-impact ROI’s. The 
literature review will be presented in three sections. The 
first section outlines the published body of literature 
associated with the document factor stated above; the second 
section shows the limited amount of literature available in 
the following areas: 

® Reader impressions of written communication, 


® Reader perception of the writer of a written 
communication, 


® Reader’s writing style preferences; 
and finally, the third section gives a detailed description of 
the results of several PERSEREC studies in the area of ROI 
document design, organization and writing style. Included in 
this third section, will be a discussion of the results of the 


Suchan study on which this thesis is a continuation. 


B. DOCUMENT FACTORS 
1. Readability Formulas 

Readability formulas are important because for decades 
researchers have used them as a dominant tool for measuring 
document effectiveness. This study avoided using these 
formulas to determine the relative readability and efficiency 
of current ROI’s because of their significant limitations. 
One particular researcher (Seltzer, 1981) summarizes their 
limitation the best. 

Seltzer argues that because readability formulas only 
assess sentence length and word choice, they fail to account 
for important reader comprehension factors. These factors 
include: 

@® the reader’s interest in the document; 


@® the reader’s familiarity with a specialized language that 
might be used in a document; 


® the effect that convoluted sentences, poorly placed 
modifiers, and other syntactic difficulties have on a 
reader’s ability to process a sentence; 


@® the reader’s ability to see easily logical association 
between groups of sentences and paragraphs. 


Another important research study (Hirsch, 1977) points 
out other flaws in these formulas. Hirsch argues that 
readability formulas fail to consider several important 
factors that contribute to a document’s effectiveness. He 
believes that to make a document readable and easy to 


understand, the writer must use organizational, syntactic, and 


linguistic strategies that will constrain meaning. This is 
accomplished by reducing the semantic-syntactic possibilities 
that can occur in a clause, sentence, paragraph, or even a 
document. Hirsch also contends that subject-verb patterns, 
the voice of verbs, clause length, and transitional tags can 
be used in conjunction with each other to improve document 
processing speed and comprehension. 

It is clear that there are severe limitations to using 
readability formulas when assessing the effectiveness of a 
document. The above two researchers point out their 
Significant flaws and their argument against these formulas. 
However, they only allude to the specific factors that make a 
document effective. 

Those specific factors include subject-verb-object; 
transitional tags and internal previews; active verbs; 
shortened paragraphs, sentence and clause length; and several 
others. These factors were used to generate the revised high- 
impact document in the Suchan Study. The remainder of this 
section will be devoted to outlining the literature that 
describes the above stated document effectiveness factors. 

2. Document Organization 

Several business communication researchers have made 
Significant discoveries about the factors that influence 
information processing. Two particular researchers (Fielden 


& Dulek, 1984) have studied and assessed document 


10 


effectiveness from a macro-organizational perspective. They 
argue that a "bottom line" communication technique decreases 
reading time and improves document comprehension. The essence 
of this technique is for the writer of nonsensitive documents 
to state their purpose for writing in the first paragraph. 

These researchers further argue that to decrease 
reading time and improve comprehension in longer documents, 
the writer should also include a contract sentence immediately 
following the bottom line. The contract sentence, basically 
an internal preview, organizes the remainder of the 
communication for the reader by providing a skeletal framework 
of the major point to be covered and the order in which they 
will be presented. 

Although Fielden and Dulek cite no empirical evidence 
to support their claims about the advantages of bottom lines 
and contract sentences, the work of two cognitive 
psychologists (Bransford & Johnson, 1972 and 1973) justify the 
value of these two organizational strategies. Bransford and 
Johnson found that subjects who were provided a concrete title 
to a very general, context-free, written passage could better 
understand and remember the passage than those not provided a 
title. The title provided a mental model that helped readers 
classify and integrate information. Similarly, a bottom line 
and contract sentence may help a reader create a mental model 


of a document’s content and constrain its meaning. 


ay. 


Another study supports Fielden’s and Dulek’s findings. 
This study  (Suchan, 1989) assessed the communication 
efficiency of the high-impact style versus the traditional 
bureaucratic low-impact style favored by the government. The 
Study involving 262 naval officers, and Suchan found that the 
high-impact style took the officers between 17% and 23% less 
time to read than the bureaucratic style, and produced much 
better message comprehension. 

3. Informational Design 

Another document factor that determines its 
effectiveness is visual design. Visual design includes the 
writers use of headings, list, bold print, underscoring, 
Spacing, and so on. Several researchers (Kostelnick, 1988; 
Hartley & Trueman, 1985; Battison & Landesman, 1981) have 
confirmed that using these strategies improves effectiveness. 

Kostelnick found that local design strategies (lists, 
bullets, indentation, and so on) may restrain meaning by 
enabling readers to code and chunk information more easily. 
Hartley and Trueman have proved through extensive testing that 
headings improve reader performance. Battison and Landesman 
have shown that FCC regulations revised into a more visually 
appealing question-answer format improved reader 


comprehension. 
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4. Sentence Level Research 

Three researchers (Miller, 1956 and 1970; Bever, 1972; 
Clark & Clark, 1968) have studied how readers use short and 
long term memory to process sentences. In particular, Clark 
and Clark found that because a reader’s short-term memory can 
hold about seven chunks of information, readers confronted 
with too much modifying detail (left embedding of 
information), may have difficulty retaining in short-term 
memory the primary subject-verb unit of the sentence. 

The relevance of this research is that it justifies 
the need for writers to create word sequences that are easy 
for readers to form into stable groups. A writer should use 
the subject-verb-object order throughout the document to 
create a stable, predictable pattern that places little demand 
on the reader’s short-term memory. The use of these sequences 
should increase reading speed and improve comprehension. 

5. Active-Passive and Word Choice Research 

Three studies (Olson & Filby, 1972; Danks & Sorce, 
1973; Charrow & Charrow, 1979) have shown that readers process 
active verbs faster than passive ones. This is because 
passive verbs invert the typical subject-verb-object order 
syntactic pattern. 

Finally, research (Rosch, 1973 and 1975) has shown 
that we are able to process quickly and accurately language 


that represents basic classifications or prototypes. Rosch 
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found that language which is relatively concrete (e.g. "cow" 


versus "bovine") can trigger other clarifying word (abstract 
background information) stored in long-term memory. What 
constitutes concrete language, however, may be 


organizationally unique. 

For example, adjudicators are accustomed to reading 
specialized terms such as "PSI" and “ROL: These terms 
trigger a range of concrete associations in the adjudicator 
but not in a reader unfamiliar with security issues. 1i9) 
short, the consistent use of context-based, organizationally 
specific specialized terms does not degrade comprehension. 

These results support the assertion that if a writer 
does not use concrete language as defined by the 
organizational context, the reader may not see recognizable 
prototypes and thus must attempt to construct them while 
reading. This construction process increases reading time, 
strains short-term memory, reduces comprehension, and degrades 


document effectiveness. 


C. READER PERCEPTION, IMPRESSION, PREFERENCE RESEARCH 

An exhaustive on-line computer search was conducted to 
determine the extent of prior empirical research on readers 
perception of writers based on writing style and document 
design. Only 3 studies were located. 

Brown and Herndl (1986) found that upper-level managers in 


a large engineering firm perceived reports written in a 
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bureaucratic style to be more competent and objective than 
reports uSing plain language guidelines that the company had 
recently implemented. Rogers (1989) discovered that automotive 
managers strongly resisted using company plain language 
criteria in their dealer contact reports because they believed 
they would be perceived by their bosses as being less thorough 
and analytical. 

Another Study (McClure, ISS G)) surveyed the lower 
management levels of Sprint/United Telephone of Florida to 
determine if these managers perceived their reading 
preferences to be the same as those of their upper-level 
Hemagers. McClure found that employees were more aware of 
their boss’s reading preferences than their CEO’s. Also, as 
communication flowed to higher levels of management, the 
employee wrote longer sentences, used bigger words, and relied 
on the passive voice in order to appear more _ formal, 
respectful, and educated. These choices led to a increased 


reading difficulty. 


D. PERSEREC STUDIES 

As shown in the earlier sections of this chapter, there 
exists a significant body of literature that defines the 
Qualities of a clearly written document. PERSEREC has 
conducted three studies (Haag, Schroyer & Crawford, 1989; 
Suchan, 1989 and 1992) that relate to adjudicators’ ability to 


process and assess information. 
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1. Haag, Schroyer, and Crawford Study 
Haag, Schroyer, and Crawford found” that adjudicavem. 
are not satisfied with the way information is presented and 
that they desired the reports be written more clearly. They 
also concluded that adjudicators had difficulty locating and 
extracting information from narrative segments of ROI’s. The 
adjudicators hoped that the ROI could be formatted to parallel 
and Support the adjudication decision-making process. 
2. Suchan (1989) Study 
Similarly, Suchan also found that the adjudicators 
were dissatisfied with the writing style of the ROI. In his 
other 1989 study, he used talk-aloud protocols and field 
interviews to discover that adjudicators had difficulty 
sorting and classifying information in the narrative sections 
Of ROI’s with Significant derogatory information. 
Adjudicators often had to reread the report sections, take 
extensive notes, and use other strategies to create the 
information patterns necessary for them to apply the 
adjudication criteria. This 1989 study was the precursor to 
the 1992 Suchan study on which this thesis is based. 
3. Suchan (1992) Study 
Suchan’s primary objective in, "The Comparison of High 
and Low-Impact Report of Investigation on the Adjudicators’ 
Decision Making: A Pilot Study", was to determine whether 


adjudicators assess ROI’s written in a high-impact style 
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differently from the same ROI’s written in the current low- 
impact style. Secondly, he examined if adjudicators at 
various central adjudication sites assessed differently the 
high and low impact ROI’s. Furthermore, he determined if 
adjudication site affected the degree to which adjudicators’ 
decisions conformed to those of their superiors. 

Suchan’s results contradict previous’ laboratory 
research that show clear writing, as exemplified by the high- 
impact style, helps readers to more effectively process 
documents. The data show there are no statistically 
Significant differences in decision type between the high and 
low impact ROI’s. Also, high-impact ROI’s did not result in 
decisions that more closely matched those of supervisory 
adjudicators, except at the Army adjudication site. In short, 
the high-impact ROI’s did not help adjudicators make better 


quality decisions. 
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ad ee RESEARCH METHODOLOGY 


A. CHAPTER INTRODUCTION 

This chapter gives a detailed outline of the research 
methodology developed and used to gather data for this study. 
It also describes the techniques used by the author to analyze 
the remaining three parts of the questionnaire. The topics 
included in this chapter are as follows: case selection, 
content and treatment; description of the adjudicator sample 
population; testing procedures; questionnaire content; 
statistical methods for analyzing the questionnaire results; 
organization of analysis; and the limitations of the 


methodology and analysis. 


B. CASE SELECTION, CONTENT, AND TREATMENT 
1. Case Selection 

Earlier PERSEREC research (Wiskoff and Fitz, 1991) 
found that adjudicators dealt with an average of 1.7 issues 
per case and assessed cases with multiple issues 46.3% of the 
time. Consequently, cases that contained two issues were used 
in this study. 

Furthermore, research shows that adjudicators most 
often dealt with Financial, Drugs/Alcohol, and 
Emotional/Mental issues. One of the most frequent double issue 


combinations was drug and alcohol (Lewis, Koucheravy,& Carney, 
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mao, Wiskoff & Fitz, 1991). 

With this in mind, multiple issue cases containing 
combinations of Financial, Drugs/Alcohol, or Emotional/Mental 
derogatory information were requested from the Personnel 
Investigation Center (PIC). After careful reading of (56) 
micro-fiche cases that met these criteria, two cases were 
chosen that best met the multiple issue and issue frequency 
@riteria. These cases also reflected the document design, 
organizational, and stylistic characteristics typical of ROI’s 
containing derogatory information (Suchan, 1989). 

2. Case Content 

The two cases selected are called Czarnek and Rokitka, 
the pseudonyms for the subject in each case. A brief overview 
of each case is described below. 

a. Czarnek 

Czarnek is a 24 year-old male applying for a Top 
Secret clearance required for a security guard position. The 


major derogatory information in his case includes: 


® falsification of his PSQ, 

@® aberrant behavior ying personal threats, and 
insubordination) resulting in an honorable discharge from 
the military after only 5 weeks in service, 

® personal and on-the-job dishonesty, 


® minor criminal conduct resulting in loss of employment, 


® minor financial matters 
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b. Rokitka 
Rokitka is a 49 year-old female applying for a Top 
Secret clearance which is necessary for her to work as a 
computer programmer. The major derogatory information in her 
case includes: 


@® alcohol/drug abuse: use of marijuana, cocaine, and 
barbiturates 


® criminal conduct: use of cocaine on the job 


® falsification of PSQ 


3. Case Treatment 
Suchan revised both the Czarnek and Rokitka cases 
using document design, organizational, and stylistic 
strategies that research has demonstrated leads to improved 
comprehension (Duffy, 1985; Redish, 1989; Seigel, 1978; 
Selzer, 1983; Suchan, 1989). Furthermore, a thorough 
literature review was conducted to ensure that each of the 
independent variables used in revising the low-impact ROI’s 
improved comprehension. These independent variables include: 
® Paragraph Length: paragraphs were no longer than 3-4 
sentences; one sentence paragraphs were used to emphasize 
important information (Felker, Redish, Peterson, 1985). 
® Headings and Subheadings: major headings and subheadings 
were used to telegraph major sections of derogatory 
information (Redish,1989). 
® Lists and Bullets: these graphic aids highlighted 
Significant information, broke OVhe Statistical 
information, and laid out complicated chronological 


sequences (Benson, 1985; Rubens, 1986). 


® Internal Previews: at the beginning of long derogatory 
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sections, an internal preview was used to provide a quick 
Overview of the major points that were to come in that 
section. The language in the preview mirrored the language 
in the major headings (Redish, Battison, Gold, 1985). 


e Clauses in Subject-Verb-Object Order: subjects, verbs, and 
objects within clauses were kept as close to each other as 
possible to avoid strain on short-term memory (Fodor & 
Garrett, 1967). 

® Modifier Strings Right Branched: long strings of modifying 
information were placed to the right of the object (right 
branching) versus before the subject (left branching). 
Research has shown that right branching of information 
enables readers to read faster and remember more easily 
information (Haviland & Clark, 1974; Miller & Isard, 
1964). 

® Active Verbs: when appropriate, passive verbs were changed 
to active verbs to make clear the agent of a particular 
action (Mirel, 1988; Selzer,1983). 

® Explicit Transitional Tags: clear transitional tags, often 
left out in narrative report writing, were added to make 
it easier for readers to quickly grasp relationships 
between ideas and sections of reports (Guillemette, 1987). 

Only material field agents had written was revised. 
Information contained in subjects’ and references’ statements, 
medical assessments, and military reports remained as 
originally written. Also, two experienced field agents 
reviewed the revised cases and noted that though the cases 
"looked" different, there were no differences in content. 

As indicated in the Introduction Chapter, the revised 


cases are called "high impact" and the original unmodified 


cases "low impact." 
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C. DESCRIPTION OF SAMPLE POPULATION 
Forty senior adjudicators (CS li through GS™1i3)™trom feu 
major central adjudication sites participated in the study. 
Adjudication supervisors at each site chose the adjudicators 
to be tested. Once again, the breakdown of adjudicators from 
each site is listed below: 
® Army Central Adjudication Facility (12) 
® Navy Central Adjudication Facility (12) 
® Air Force Security Clearance Office (AFSCO) (4) 
® Directorate for Industrial Security Clearance Review 
(DISCR) at Celumbus sae 
A high caseload backup prevented AFSCO from allowing more 
adjudicators to participate the study. 
A total of 31 female and 9 male adjudicators read the 
cases. Average age was 45, and average years of adjudication 
experience was 13. The typical adjudicator had spent slightly 


more than 6 years at the adjudication site. 


D. TESTING PROCEDURES 
The adjudicators were tested on site. At each site they 
were divided randomly into two groups, with each group 

adjudicating two cases. 
® Group I (six adjudicators at each site except for AFSCO) 
received a revised, high impact treatment of the Czarnek 
case and an original, low impact treatment of the Rokitka 


Case. 


® Group II received a low impact treatment of the CzZarnek 
case and a high impact treatment of the Rokitka case. 


22 


A counterbalanced 2 X 2 design was used to ensure that 
eee adjudicator responded toya different case written in the 
typical, low-impact style and the revised high-impact style 
and that order of presentation would not influence results. 

At each adjudication site, the adjudicators assessed the 
cases in a large room. To guarantee that all adjudicators were 
given the same instructions, the researcher read a prepared 
script that explained the purpose of the exercise, the 
materials they were receiving, and process they were to use in 


assessing the cases and completing the questionnaires. 


E. CONTENT OF QUESTIONNAIRE 

Adjudicators were asked to complete a questionnaire after 
adjudicating each case. The first questionnaire contained 
three parts: 


® Part I contained seven open-ended questions, requesting 
information about the adjudication decision, the rationale 
for the decision, the adjudication criteria applied, the 
mitigating criteria applied, the need for more 
information, and the perceived need to send a Letter of 
Pibent (Ol) . 


® Part II contained 23 questions about the readability of 
the case, the ease in finding derogatory information, the 
quality of field agents’ writing, and so on. A five-point 
bipolar scale was used to capture these perceptions. See 
Appendix A. 


® Part III examined adjudicators’ perceptions or feelings 
toward the field agents who wrote the ROI. Fifteen 
semantic differential items on a five-point scale were 
used to capture this data. See Appendix B. 
The second questionnaire, completed after the second case 


was adjudicated, was identical to the first except it 
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contained an additional fourth part (see Appendix C) that 
asked adjudicators which case was easier to read, required 
less rereading, increased confidence in the competence of 
field agents, and so on. There were 8 case preference 
questions of this type and two yes/no questions. 

This thesis focuses on the results from Part II, III, and 


Iv of the questionnaire. 


F. STATISTICAL TESTING OF QUESTIONNAIRE RESULTS 
1. Software Statistical Package 
The questionnaire results from the Suchan Study were 
recorded for analysis in statistical package called APL. This 
package was used to analyze the results from Part I of the 
questionnaire. The analysis of the remaining parts of the 
questionnaire were completed using the statistical package 
SPSS. Much care was taken when transferring the questionnaire 
result from APL to SPSS. The first part of the survey was 
re-analyzed using SPSS and compared with the previous APL 
results. Since the result were the same, the integrity of the 
data was maintained. 
2. Confidence Interval 
Since this is a pilot study of how document design, 
organization, and writing style impact upon the impressions, 
perceptions, and preferences of adjudicators, a 5% confidence 
interval was determined as significant during statistical 


analysis of both the high and low-impact Czarnek and Rokitka 


Cases. A 10% confidence interval was deemed sufficient when 
the cases were analyzed independently, or the population of 
the adjudicators was stratified. With this in mind, the 10% 
confidence interval results from our analysis need to be 
interpreted with care. 
3. Statistical Techniques 
a. ANOVA’S 

When analyzing results of Part II and III of the 
questionnaire, ANOVA’s were used for determining the 
statistical significance of the comparative data from the 
high-impact and low-impact reports. The dependent variables 
were the specific factors coded into the questions themselves, 
while the independent variable in all tests was the specific 
report writing style (i.e. high-impact or low-impact). An 
example of a dependent variable from Part II would be ease of 
reading the subject interview section. 

b. Chi-Squares 

When analyzing results of Part IV of the 
questionnaire, Chi-Squares were used for determining the 
statistical significance of the comparative data from 
respondents reading the high-impact and low-impact reports. 
Once again, the dependent variables were the specific factors 
coded into the questions themselves, while the independent 
variable in all tests, was the specific report writing style 


(i.e. high-impact or low-impact). 
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Of the ten questions in Part IV, eight were 
analyzed using this technique. Questions 6 and 8 were not of 
comparative nature (i.e. choosing between Case I or Case II), 
but were yes/no questions. Because of this yes/no design, 
there waS no way to tie these questions to the independent 


variable, which was the effect of the writing style. 


G. ORGANIZATION OF ANALYSIS 
The analysis of each part of the questionnaire was to 
determine the statistical significance between the question 
responses for the high-impact versus the low-impact ROI’s. The 
key difference throughout the analysis was the stratification 
of the adjudicator population and/or the specific case 
treatment analyzed. This breakdown is as follows: 
® Both Cases: The high/low impact Czarnek and Rokitka 
high/low-impact cases were analyzed using the entire 


population of adjudicators. 


® Czarnek Cases: The high/low-impact Czarnek cases were 
analyzed using the entire population of adjudicators. 


@® Rokitka Cases: The high/low-impact Rokitka cases were 
analyzed using the entire population of adjudicators. 


@® Army Adjudication Site: The high/low impact Czarnek and 
Rokitka high/low-impact cases were analyzed using the 
result from just the Army population of adjudicators. 


@® Navy Adjudication Site: The high/low impact Czarnek and 
Rokitka high/low-impact cases were analyzed using the 
result from just the Navy population of adjudicators. 


@® DISCR Adjudication Site: The high/low impact Czarnek and 


Rokitka high/low-impact cases were analyzed using the 
result from just the DISCR population of adjudicators. 
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@® Demographic Breakdown: The high/low impact Czarnek and 


Rokitka high/low-impact cases were analyzed using a 
demographic breakdown of the adjudicator population. A 
more detailed description of the demographic breakdown is 
given below. 


DESCRIPTION OF DEMOGRAPHIC BREAKDOWN 


The questionnaires were analyzed using a demographic 


breakdown of the adjudicator population. The following four 


demographic categories were used for this analysis: 


Age: The first demographic category was age, young and 
Old. Young was considered below one standard deviation 
From the mean adjudicator age, while old was everyone 
else. The mean age of adjudicators was 44.2 years, one 


Standard deviation was 10.4 years, so the cut-off for the 
young category, was 33.8 years old. Under this criteria, 
three (3) adjudicators were fell into the young category, 
while (36) fell into the old. 


sex: The second category was sex, male and female. The 
sexual breakdown was nine (9) male to (31) female 
adjudicators. 


Grade Level: The third category was grade level. The 
population was stratified between GS-11 versus GS-12 and 
GS-13 adjudicators. Under this criteria, nine (9) fell 


into the GS-11 category, (28) into the GS-12 and GS-13 
category, and three (3) adjudicators not listing their 
grade level on the questionnaire. 


Years in Security Administration Service: The final 
category was the number of years of service that a 


adjudicator had worked in security administration. The 
population was stratified between three years or less 
experience, versus greater that three years experience. 
Under this criteria, three (3) adjudicator fell into the 
Prior Category and (37) into the latter. 
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I. LIMITATIONS 

The sample size of some of the adjudicator population 
breakdowns limited the strength of the analysis. 
Specifically, the cell sizes were (n=6) when we compared case 
treatment and decision type within adjudication sites, and as 
small as (n=3) for two demographic population breakdowns. 
Consequently, the data in these report sections must be 
interpreted with care. 

Also, Suchan’s Study tested the adjudicators in a quiet 
room, free from distractions. This environment does not 
replicate actual workplace conditions. Adjudicators work in 
busy offices where numerous distractions and frequent 
interruptions can make reading long ROI’s difficult. Because 
of this atypical testing environment, the questionnaires 
results do not capture all of the high-impact style’s possible 


reading ease and information retention advantages. 
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Iv. PRESENTATION AND ANALYSIS OF ADJUDICATION TIMES AND 
THE ADJUDICATOR’S IMPRESSIONS OF CASE INFORMATION 

This chapter analyzes both the times adjudicators required 
to assess the high and low-impact cases as well as their 
responses to Questionnaire Part II (see Appendix A). The 
chapter is organized in three sections; the first two sections 
present and analyze the adjudication times and Questionnaire 
Part Il respectively, while the final section draws 


eouclusions. 


A. PRESENTATION AND ANALYSIS OF THE TIME REQUIRED TO 

ADJUDICATE BOTH THE HIGH AND LOW-CASES 

These results indicate that there was no statistically 
Significant differences between the time required to 
adjudicate the high-impact Czarnek and Rokitka cases versus 
their low-impact counterparts. The mean times to adjudicate 
both the high and low-impact cases are 18.33 and 20.08 minutes 
respectively. The corresponding standard deviations are 6.51 
and 8.54 minutes. The resulting value of the F-statistic is 
1.046, which shows a significance of 31%. This significance 
level is well outside of the 5-10% criteria. 

The analysis results from the Czarnek and Rokitka cases, 
each adjudication site, and most of the demographic 


breakdowns, also produced no statistically significant 
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differences, with one exception: the male adjudicator 
population. 

The male adjudicator results, however, indicate that there 
was a Statistically significant difference between the time 
they required to adjudicate the high-impact Czarnek and 
Rokitka cases versus their low-impact counterparts. The mean 
times to adjudicate both the high and low-impact cases are 
13.5 and 21.75 minutes respectively. The corresponding 
Standard deviations are 1.91 and 12.2 minutes. The resulting 
value of the F-statistic is 3.98, which shows a significance 
of 6.6% and satisfies the 10% confidence criteria. 

Overall, adjudication testing time results indicate that 
the high-impact design, organization and writing style does 
not significantly decrease the time required to adjudicate 


Cases? 


B. PRESENTATION AND ANALYSIS OF QUESTIONNAIRE PART II RESULTS 

This section of the chapter presents and analysis the 
result to the questionnaire which tested the adjudicators’ 
impressions of case information. 

As stated in the first and third chapter, Questionnaire 
Part II tests adjudicators’ impression concerning case 
information. Each of the twenty-three questions in this part 
of the survey asked adjudicators to rate a specific aspect 
concerning the presentation of case information such as its 


readability and the ease of finding derogatory information. 
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A five-point bipolar scale was used to capture results. 
Specifically, this part of the survey was designed to 
determine if there were significant differences between the 
response to the questions for the high and low-impact cases. 
The first subsection gives direction on how to read the 
tables of results, and the last draws conclusions. The 
remaining middle subsections are organized as follows: 


® Both Cases: Analysis of both Czarnek and Rokitka high/low 
impact cases using entire adjudicator population. 


® Czarnek Cases: Analysis of Czarnek high/low impact cases 
using entire adjudicator population. 


® Rokitka Cases: Analysis of Rokitka high/low impact cases 
using entire adjudicator population. 


® Separate Adjudication sites: Analysis of each Adjudication 
Site (Army, Navy, and DISCR) using both cases. 


® Male Adjudicators: Analysis of both cases using Male 
Adjudicators. 


1. Directions for reading tables of results 
There are seven tables of results in this chapter. 
Fach table has a title block and seven columns of information. 
Below iS an extract of the title block and column heading 


with an explanation of each. 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 
CASES AND ADJUDICATORS TESTED (n = sample size) 
ADJUDICATOR’S IMPRESSIONS OF CASE INFORMATION 
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The title block is the same for all tables with the 
exception of the second line, which is highlighted above. 
This line states which cases and adjudicators were included 
for the analysis. An explanation of the seven columns 
headings are as follows: 

@® QUEST #: The Part II question number analyzed. 


® HI MEAN: The mean score on a five point bipolar scale for 
the high-impact cases. 


® HI S.D.: The standard deviation of the high-impact score. 


® LI MEAN: The mean score on a five point bipolar scale for 
the low-impact cases. 


® LI S.D.: The standard deviation of the low-impact score. 


® F-STAT: The value of the F Statistic resulting from the 
ANOVA testing. 


® SIG. of F: The significance of the F Statistic, which is 
the probability that the resulting F value is the due to 
a random occurrence. 

The results of questions which fall within the 5% 
confidence interval are shaded while the result that meet the 
10% interval are highlighted in bold type. 

2. The Analysis of both the Czarnek and Rokitka Case 

Results 

The first primary research question, outlined in the 
first chapter, asks 

@® Will there be statistically significant differences 
between adjudicators’ impressions of their ability to 
process case information presented in a high-impact ROI 


versus a low-impact one? 


The answer to that question is given in Table 4-1. Out of the 
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results to twenty-three questions, thirteen were significant 
within 5% while another four were significant within 10%. 
Those questions that met the 5% criteria will be presented in 
the following subsections. 
a. Questions Statistically Significant Within the 5% 
Confidence Interval 
(1) Question 1: Coverage of Issues 

This question asks to what extent does the case 
provide good to poor coverage of issues to enable the 
adjudicator to make a high quality adjudication decision. The 
mean score for the high-impact cases (1.9) falls just to the 
left of the somewhat good rating, while the mean score (2.4) 
for the low-impact cases lies between the somewhat good and 
neutral rating. These results indicate that adjudicators feel 
that the high-impact style provides more thorough coverage of 
the case issues than the low impact style. 

Since each case contained the same content, the 
design, organization, and style of the high-impact ROI 
affected adjudicators’ perception of the document’s 
thoroughness. 

i  wOuecstien 2; Presentation of Derogatory 
Information 

This question asks if the way a field agent 

presents derogatory information in a case makes it easy or 


feet icult to apply the adjudication criteria to that 
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TABLE 4-1 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 


ANALYSIS OF BOTH CZARNEK & ROKITKA CASES (n = 40) 
ADJUDICATOR’S IMPRESSIONS OF CASE SENEQRMAT LON aa 
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information. The mean score for the high-impact cases (1.95) 
lies just to the left of the somewhat easy rating, while the 
low-impact scores (2.46) falls between the somewhat easy and 
neutral rating. The results show that the high-impact style 
makes it easier to apply the adjudication criteria. 

(3) Questions 5, 6, and 7: Perception of Field 
Agents Awareness 

Questions 5, 6 and 7 gauge the adjudicators’ 

perception of field agents’ awareness of their needs. 
Specifically, the questions asked the adjudicators, based on 
the way information was presented in the cases, did they feel 
the field agent seemed to be aware or unaware of the 
following: 


® Question 5: The information needed to make a high quality 
adjudication decision; 


® Question 6: How to make the case easier to read; 


® Question 7: The large amounts of daily reading done by the 
adjudicators. 


The high-impact cases mean scores are 1.73 for question 5, 
2.05 for question 6, and 2.03 for question 7. These scores 
all fall within the area of the "somewhat aware" rating 
Category. In contrast, the low-impact cases mean scores are 
fetomeror guestien 5, 2.58 Eor questiomes6, and 2.63 for 
question 7. These results lie between the somewhat aware and 


neutral rating categories. 
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Although the adjudicators perception of the 
field agents who wrote the cases are tested much more 
thoroughly in Questionnaire Part III, the results of these 
three questions point out that document design, organization 
and writing style do affect readers’ perception of the 
writer’s awareness. The author of a high-impact style 
document is perceived as having a greater awareness of the 
reader’s needs than the author of a low-impact one. 

(4) Question 8: Evaluation of Narrative Report 
Writers 

Question 8 asked adjudicators if they believed 
that the field agents who wrote the narrative reports for the 
cases were good or poor writers. The high-impact mean score 
(1.93) falls just to the left of the somewhat good rating, 
while the low-impact score (2.5) lies exactly in the middle of 
the somewhat good to neutral rating. The result to this 
question showed the greatest statistical significance. They 
are Significant at the .6% confidence level. These results 
Show that the majority of adjudicators believe that the 
authors of the high impact narrative reports are better 
writers than the authors of low impact reports. 

(5) Question 11, and 12: Ease of Finding and 
Reading Derogatory Information 
Questions 11 and 12 tested the ease of finding 


and reading derogatory information in the cases. The high- 
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impact mean scores are 1.9 for question 11 and 1.83 for 
@eestion i2, which both fall to the left of the somewhat easy 
rating. The low-impact mean scores are 2.5 for question 11 
and 2.33 for question 12, which both fall between the somewhat 
easy to neutral rating. 

These results indicate that adjudicators found 
the high-impact style cases made finding and reading 
information easier. Some of the specific characteristic of a 
high-impact document that make finding and reading information 
easier include the use of 

@® heading and subheadings 

@® lists and bullets 

® internal previews 

® active verbs 

® and explicit transitional tags. 

The results show that these and other high- 
impact stylistic strategies created the perception that 
finding and reading derogatory information was easier. 

(6) Questions 14, 16, and 19: Ease of Remembering, 
Locating and Following Presented Case Information 

These questions asked the adjudicators the 

relative ease they had in the following areas: 
® Question 14: remembering the presented case information, 


® Question 16: locating the essential information necessary 
for the efficient adjudication of the case. 
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@® Question 19: following the chronological sequence of 
events in the case. 


The high-impact cases mean scores are 1.9 for question 14, 1.8 
for question 16, and 2.05 for question 19. These scores all 
fall in the area of the somewhat easy rating category. The 
low-impact cases mean scores are 2.5 for question 14, 2.48 for 
question 16, and 2.6 for question 19. These results cluster 
between the somewhat easy and neutral rating categories. 

The results show that adjudicators feel that 
information presented in a high-impact style is easier to 
remember and locate than information presented in low impact 
Style. They also indicate that the high-impact style makes it 
easier to follow chronological sequences of events. This can 
be interrupted to mean that the high impact style increases 
information retention in short term memory, improves 
efficiency in locating essential information in the document, 
as well as improve their ability to follow a sequence of 
events. 

(7) Question 15: Ease of Adjudicating Case 

The adjudicators were asked in this question 
whether they found it easy or difficult to adjudicate the 
case. The high-impact mean score (1.78) falls just to the 
left of the somewhat easy rating, while the low-impact score 
(2.3) lies between the somewhat easy to neutral rating. 

The result amplifies the finding to questions 


14, 16, and 19. Ease in locating and remembering essential 
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case information, and following chronological events should 
leads to the easier adjudication of cases. 
(8) Question 22: Fatigue of Reading 

Question 22 asked the adjudicators whether they 
agreed or disagreed with the statement "Reading this case 
bared me out." The high-impact mean score (4.38) falls 
between the somewhat disagree and disagree rating categories, 
while the low-impact score (3.78) lies between the neutral 
rating and the somewhat disagree ratings. 

The above results indicate that the high-impact 
cases caused less fatigue than the low-impact ones. The high- 
impact document factors that may contribute to this decreased 
fatigue, include the use of headings and subheadings, as well 
as bullets and lists. 

b. Questions Statistically Significant within the 10% 
Confidence Interval 

Although the results to questions 3, 4, 18, and 23, 

Gid not meet the 5% criteria, they did show significance 

within the 10% confidence interval. Questions 3 and 4 tested 

the adjudicators confidence in the quality of the field 

agents’ investigation and the soundness of their adjudication 

decision. The frequency of rereading sentences was the 

objective of question 18, while eye strain was the factor 


tested for in question 23. 
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The high-impact cases mean scores are 1.93 for 
question 3, 1.38 for question 4, 3.88 for question 18, and 3.7 
For question 23. The high-impact scores for both questions 3 
and 4, fall between confident and somewhat confident rating 
category, question 18 scores lie between the somewhat never to 
never rating, and question 23 scores fall between the somewhat 
disagree to disagree ratings. 

The low-impact cases mean scores are 2.35 for 
question 3, 1.65 for question 4, 3.45 for question 18, and 3.1 
for question 23. The low-impact scores for questions 3 fall 
between the somewhat confident and neutral rating category, 
question 4 falls between the confident and somewhat confident 
ratings, question 18 scores lie between the somewhat never to 
never rating, and question 23 scores fall between the somewhat 


disagree to disagree ratings. 


In all of the above questions, it is inferred that 
the design, organization, writing style of the high-impact 
cases increased 1) the confidence in the field agent’s 
investigation and adjudicators’ decision, 2) reduced the 
frequency of sentence rereading, and 3) reduced eye strain. 

c. Summary of Both the Czarnek and Rokitka Case 

Results 

When we combine the results of the thirteen 


questions that passed the 5% confidence test, and the four 
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that passed the 10% test, it is clear that adjudicators feel 
that the document design, organization, and writing style of 
the high-impact cases to be superior to the currently used 
low-impact style cases. As will be seen in the following 
sections, this result is not necessarily true when the Czarnek 
and Rokitka cases or the adjudications sites are analyzed 
separately. 
3. Analysis of the High and Low-Impact Czarnek Cases 

The results of the analysis of the high and low-impact 
Czarnek cases are presented in Table 4-2. Out of the twenty- 
three questions analyzed, none of the results fell within the 
5% confidence interval and only four (#’s 2, 8, 16, and 22) 
met the 10% confidence criteria. The following is a summary 
of each of the four questions: 


@® Question 2: How easy was it to applying the adjudication 
criteria to the case information? 


@® Question 8: What was the quality (good to poor) of the 
narrative report writers? 


@® Question 16: How easy was it to locate essential case 
information? 


@® Question 22: Was reading the case tiring? 

Comparing the result to these questions with those 
from the analysis of both cases, we see consistent findings. 
That means that the above statistically significant questions 
were also found significant within either the 5% or 10% 
interval in the combined Czarnek and Rokitka analysis (Table 


4-1) 
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TABLE 4-2 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 


ANALYSIS OF THE CZARNEK CASE (n = 20) 
ADJUDICATOR’S IMPRESSIONS OF CASE INFORMATION 
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Given that there were only four statistically 
Sigmericant questions overall, and all of these fell only 
faeenan the 10% interval, these results tell us that there was 
not a strong difference in adjudicators’ impressions of the 
high and low-impact Czarnek cases. It can be further inferred 
that Czarnek results are not the driving force behind the 
large amount of statistical significance found when both the 
Czarnek and Rokitka Cases were analyzed together. This is 
counter-intuitive to expected results, because the Czarnek 
case content is more complicated and longer than Rokitka case 
content. Common sense would dictate that the longer and more 
complicated a document’s content is, the more helpful the 
high-impact style would be in reading, finding and locating 
information. 

4. Analysis of the High and Low-Impact Rokitka Cases 

The results of the high and low-impact Rokitka case 
data are presented in Table 4-3. Questions 8, 12, 14, and 
[ewere found to be statistically significant within the 5¢ 
eiterval while questions 1, 5, 6, 7, 11, 15, and 16 were 
S@omiticant within the 10% interval. The following is a 
summary of each of the four questions that met the 5% 
confidence criteria: 


® Ouestion 8: What was the quality (good to poor) of the 
narrative report writers? 


® Question 12: How easy was it to read the derogatory case 
information? 
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TABLE 4-3 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 


ANALYSIS OF THE ROKITKA CASE (n = 20) 
ADJUDICATOR’ S IMPRESSIONS OF CASE INFORMATION 
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® Question 14: How easy was it to remember case information? 


@® Question 19: How easy was it to follow chronological 
events in the case? 


Questions meeting the 10% criteria asked the following: 
question 1, the coverage of issues in the case; question 5 and 
7, the field agents’ awareness of the adjudicators’ 
information needs and daily reading requirements; question 6, 
the field agents ability of make the case easier to read; and 
finally, Questions 11, 15 and 16: the ease in locating 
derogatory and essential information as well as adjudicating 
the case. 

Comparing the result to these questions with those 
from the analysis of both cases, we see consistent findings. 
In other words, the statistically significant response to 
questions listed above were also found significant within 
either the 5% or 10% confidence interval in the combined 
Czarnek and Rokitka analysis (Table 4-1). 

Comparing Rokitka case results to the Czarnek (Table 
4-3) shows that the document design, organization, and writing 
style of the Rokitka cases had a greater statistical 
Significance on the adjudicator’s impressions of both the high 
and low-impact writing style than the Czarnek Cases. This 
comparison makes clear that the Rokitka cases are the driving 
force behind the statistically significant results of the 


analysis of both cases. 
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5. Analysis of the Separate Adjudication Sites using both 
the High and Low-impact Czarnek and Rokitka Cases 
This section presents and analyzes the separate 
adjudication site results. There are three subsections in 
this section; each subsection presents and analyzes the result 
for the Army, Navy or DISCR sites respectively. 
a. Presentation and Analysis of the Army Adjudication 
Site Results 
Results from the Army Adjudication Site produced 
more Statistically significant results than the Navy and DISCR 
Sites combined. Table 4-4 shows that the results to three 
questions (#’s 1, 4, and 12) were significant within the 5% 
confidence interval, while five (#’s 6, 8, 10, 14, and 20) 
were within 10%. The following is a summary of each of the 
three questions that met the 5% confidence interval criteria. 


® Question 1: What was the quality (good to poor) of the 
coverage of issues in the case? 


® Question 4: How confident was your adjudication decision? 


® Question 12: How easy was it to read the derogatory case 
information? 


Questions meeting the 10% criteria asked about the following: 
question 6, the field agents’ ability of make the case easier 
to read; question 8, the quality of narrative report writer; 
question 10, the perceived speed of reading the case; 


questions 14, the ease of remembering derogatory information 
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TABLE 4-4 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 


ANALYSIS OF ARMY SITE (n = 12) USING BOTH CASES 
ADJUDICATOR’S IMPRESSIONS OF CASE INFORMATION 
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in the case; and finally, question 20, the frequency of 
underlining or note taking. 

Comparing the above results with those from the 
analysis of both cases using the entire adjudicator 
population, we see three key differences. The first two are 
that the results of both question 10 and 20 were found to be 
Statistically insignificant when both cases were analyzed. 
The third difference is that the result to Question 4 fell 
within the 5% confidence interval for this site but was only 
at the 10% level in Table 4-1. 

All other statistically significant results from 
the Army sites are consistent with the combine case analysis. 
That means that excluding the above exceptions, all questions 
found significant within either the 5% or 10% confidence 
interval were also significant in the combined Czarnek and 
Rokitka analysis (Table 4-1). 

We also found that there were several question that 
fell within the insignificant range in the Army analysis but 
Significant in the analysis of both cases. These results may 
be caused by the smaller sample size of the Army Adjudicator 
test population. 

We are uncertain why the Army site has both an 
abundance of significant results, and notable statistical 
inconsistences with the results of the combined Czarnek and 


Rokitka cases. 
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b. Presentation and Analysis of the Navy Adjudication 

Site Results 

The results of the Navy site analysis are presented 
in Table 4-5. Only the result to question #2 was 
statistically significant within the 5% confidence interval, 
and four (#’s 1, 7, 8, and 16) were within the 10% interval. 
Question 2 asked adjudicators if they felt it was easy or 
difficult to apply the adjudication criteria. The questions 
meeting the 10% criteria asked the following: question 1, the 
coverage of issues in the case; question 7, the field agents 
awareness of the adjudicators’ daily reading requirement; 
question 8, the quality of narrative report writer; and 
finally, Questions 16, the ease in locating essential case 
information. 

Comparing the above results with those from the 
analysis of both cases using the entire adjudicator population 
we find the results are mostly consistent. The only 
exceptions are that there are fewer statistically significant 
result in the Navy Site analysis, and questions 1,7, 8 and 16 
fell within the 10% criteria at the Navy Site but, were at the 
5% level when treatment of both cases were combined. 

c. Presentation and Analysis of ole DISCR 

Adjudication Site Results 

The result of the DISCR site analysis are presented 


in Table 4-6. This site produced the least amount of 
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TABLE 4-5 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 
ANALYSIS OF NAVY SITE (n = 12) USING BOTH CASES 
ADJUDICATOR’S IMPRESSIONS OF CASE INFORMATION 
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TABLE 4-6 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 


mer oes OF DISCR SITE (n = 12) USING BOTH CASES 
ADJUDICATOR’S IMPRESSIONS OF CASE INFORMATION 
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Statistically significant differences (i.e. two questions). 
The results to one question (#22) was statistically 
Significant within the 5% confidence interval while the other 
questions (#23) fell within the 10% interval. Questions 22 
and 23 asked the adjudicators if reading the case caused 
fatigue and eye strain respectively. Comparing the above 
results with those from the analysis of both cases using the 
entire adjudicator population, we see that the results are 
consistent. 

We must note that the results to the other twenty- 
one questions were well outside of the 10% interval. From 
this, we can infer that there is relatively little difference 
in the DISCR adjudicators’ impression of the case information 
between the two styles. This lack of difference is far 
greater than results from either the Army or Navy Adjudication 
Sites. 

6. Presentation and Analysis of the Male Adjudicator 
Results 

As described in the third chapter, an analysis Or 

several demographic breakdowns of the adjudicator population 

was performed. Similar to the results found in the analysis 

of the adjudicating times, the only group to produce a large 

amount of statistically significantly results was the male 


adjudicators. 
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A presentation of these results are in Table 4-7. The 
fesults tO nine questions (#’s 1,2, 3, 6, 7, 11, 14, 15, 16, 
and 18) were found to be statistically significant within 5%, 
amen five question (#’s 5, 8, 12, 17, and 19) within 10%. The 
following is a summary of the nine question that met the 5% 
confidence interval criteria: 


® Question 1: What was the quality (good to poor) of the 
coverage of issues in the case? 


® Question 2: How easy was it to apply the adjudication 
criteria to the case information? 


® Question 3: How confident were you in the quality of the 
field agent’s investigation? 


® Question 6: How aware were field agents in making the case 
easier to read? 


® Question 7: How aware was the field agent to the 
adjudicator daily reading requirement? 


® Question 11: How easy was it to find derogatory 
information? 


® Question 14: How easy was it to remember derogatory 
information? 


® Question 15: How easy was it to adjudicate the case? 


® Question 16: How easy was it to locate essential case 
information? 


® Question 18: How often were sentences reread? 
The questions meeting the 10% criteria asked the following: 
question 5, the field agents awareness of the adjudicators 
information needs; question 8, the quality of narrative report 
writer; questions 12, the ease of reading derogatory 


information in the case; question 17, the frequency of reading 
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TABLE 4-7 


SUMMARY OF QUESTIONNAIRE PART II RESULTS 


ANALYSIS OF MALE ADJUDICATORS (n = 9) USING BOTH CASES 
ADJUDICATOR’S . S IMPRESSIONS OF CASE _INFORMATION 
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Section in the case; and finally, question 19, the ease of 
following the chronological sequence of events in the case. 

Comparing the above results with those from the 
analysis of both cases using the entire adjudicator 
population, we once again see that the results are consistent. 
These results, however, are quite surprising considering the 


small sample size (n=9) of male adjudicators. 


C. CHAPTER CONCLUSION 

The results from the high and low-impact treatment of both 
the Czarnek and Rokitka cases point out that there is no 
Statistical evidence to show that the high-impact decreases 
adjudication time. Conversely, there is evidence that 
adjudicators feel that the high-impact style is an improvement 
over the low-impact one. Specifically, they believe that the 
high-impact style 

@® provides better coverage of issues; 


@® makes it easier to apply adjudication criteria to the case 
information; 


@® makes the field agent who wrote the case seem more aware 
of 1) how to make the case easier to read; 2) their 
informational needs; 3) the extenSive amount of their 
daily reading; 

@® makes the field agent seem like a better writer; 


® makes the derogatory information in the narrative report 
easier to find, read, remember and adjudicate; 


® makes following the chronological sequence of events in 
the case easier to follow; 


@® and causes less fatigue while reading. 


SiS) 


To a lesser degree adjudicators believe that the high- 
impact style makes them more confident in the soundness and 
quality of their adjudication decision, causes them to reread 
sentences less often, and produces less eye stain. 

We also found that there were greater statistically 
Significant differences in the adjudicators impressions of the 
case information in the Rokitka cases than the Czarnek cases. 
This is contrary to expected results because the Czarnek case 
content is more complicated and longer than Rokitka case 
content. One would expect that the longer and more 
complicated a documents content is the more helpful the high- 
impact style would be in reading, finding, and locating 
information. 

When the analysis was divided by adjudication site, the 
statistical significance was not nearly as great as the result 
from the entire population. However, this could be attributed 
to the smaller sample sizes used. The greatest statistical 
differences were found from the analysis of the Army 
adjudication site data. The Navy adjudication site data 
produced the second most statistically significant result, 
while DISCR data had the least. 

The most surprising result was found when analyzing the 
demographic breakdown of adjudicators. The male adjudicators 
found that the high-impact style decreased the time required 


to adjudicate a case. They also felt that the high-impact 
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cases were a much greater improvement over the current low- 


impact cases than their female counterpart. 


ST 


Nae PRESENTATION AND ANALYSIS OF PART III RESULTS: THE 
ADJUDICATOR’S PERCEPTION OF THE FIELD AGENT 

Questionnaire Part JG giale (see Appendix B) tests 
adjudicators’ perception of or feeling towards the field 
agents who wrote either the high-impact or low-impact ROTI. 
This survey section asked adjudicators to rate the field 
agents using fifteen bipolar adjectives such as objectivity, 
clarity of thinking, reliability, etc.. A five-point bipolar 
scale was used to capture the adjudicator’s feelings. 
Specifically, this part of the survey was designed to 
determine if the high or low-impact style caused differences 
in adjudicators’ perceptions of field agents. 

This chapter presents and analyzes the results to 
Questionnaire Part III in the same manner as the Part II 
results in the previous chapter. The first section of the 
chapter gives a very brief explanation of how to read the 
tables of results, and the last draws conclusion. The 
remaining middle sections cover the analysis results from both 
cases, the Czarnek cases, the Rokitka cases, the separate 


adjudication sites, and finally the male adjudicators. 


A. DIRECTIONS FOR READING TABLES OF RESULTS 
Similar to the previous chapter, there are seven tables of 


results in this chapter. These tables will follow the same 
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mermMat aS the previous tables (i.e. title block, column 
headings, shading of 5% results, and bolding of 10% results). 
The only differences are the information in the title block, 
the results to the specific questions, and total number of 


questions (15 vice 23). 


B. ANALYSIS OF BOTH CZARNEK AND ROKITKA CASES 
The thesis’s second primary research question asks 
@® Will there be statistically significant differences 
between adjudicators’ perceptions or feeling towards the 
field agents who wrote a high-impact ROI as compared to 
those who wrote the low-impact ROI? 
The answer to that question iS given in Table 5-1. Out of 
fifteen bipolar descriptive adjectives, only two were 
Significant at 5% level, while four were significant at the 
10% level. 
1. Questions Statistically Significant Within the 5% 
Confidence Interval 
a. Question 8: Agent Clarity of Thinking 
This bipolar adjective determined whether 
adjudicators perceive differences in the clarity of thinking 
between the field agent who wrote the high or low-impact case 
treatments. The high-impact mean score (1.9) falls just to 
the left of the somewhat clear thinking rating, while the low- 


impact score (2.25) lies between the somewhat clear thinking 


and the neutral rating. The results indicate that the 
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TABLE 5-1 











SUMMARY OF QUESTIONNAIRE PART III RESULTS 
ANALYSIS OF BOTH CZARNEK & ROKITKA CASES (n = 40) 
ADJUDICATOR" S PERCEPTION OF PIELD AGENT WRITING ROI 
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adjudicators felt the writer of the high-impact ROI is a more 











lucid thinker than his low-impact counterpoint. This result 

points out that the ROI document deSign, organization and 

writing style projects a more intelligent image of the writer, 

which could increase the writer’s credibility with the reader. 
b. Question 10: Efficiency of the Field Agent 

This bipolar adjective determined whether 


adjudicators perceived differences in efficiency between the 


60 


field agent who wrote the high or low-impact cases. The high- 
impact mean score (1.85) falls just to the left of the 
somewhat efficient rating, while the low-impact score (2.3) 
lies between the somewhat efficient and the neutral rating. 
The results show that adjudicators perceive the 
high-impact writer as more efficient than the low-impact 
author. This perception of efficiency could also increase the 
field agent’s credibility with the adjudicators. In other 
words, the more efficient the field agent seems to the 
adjudicators, the less likely they will feel that the 
Personnel Security Investigation (PSI) was conducted in a 
shoddy, haphazard manner. These results also suggest that 
adjudicators view the document factors in the high-impact 
Style as a more efficient means of conveying derogatory 
information than the low-impact style. 
2. Questions Statistically Significant Within the 10% 
Interval 
The results to the rating of the bipolar adjectives 5, 
6, 7, and 14 were found to be significant within the 10% 
confidence interval. These bipolar adjectives determined 
whether the adjudicators perceived differences in the field 
agents in the areas of organization, precision, care, and 
judgement respectively. 
The high and low-impact scores and their relative 


position on the rating scale are as follows: 
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@® #5: The high-impact score (1.97) falls just to the left 


of the somewhat organized rating. The low-impact score 
(2.4) lies between the somewhat organized and neutral 
rating. 


e #6: The high-impact score (1.98) falls just to the left 


of the somewhat precise rating. The low-impact score 
(2.33 lies between the somewhat precise and neutral 
rating. 


e #7: The high-impact score (1.83) falls to the left of the 
somewhat careful rating. The low-impact score (2.21) lies 
between the somewhat careful and neutral rating. This 
results shows a significance almost within the 5% 
confidence interval (i.e. 5.5%) 

@ #14: The high-impact score (1.74) falls to the left of 
the at fairly sound judgment rating. The low-impact score 
(2.4) lies between the fairly sound judgment and neutral 
rating. 

Although these results are not a conclusive as the 
findings from in the 5% interval, they indicate that the 
writers of high-impact ROI’s are regarded as more organized, 
precise, careful, and possess better judgment than their low- 
impact counterparts. 

3. Summary of Both the Czarnek and Rokitka Case Results 

When we combine the results from the rating of the two 
bipolar adjectives that passed the 5% confidence test, and the 
four that passed the 10% test, it is fairly clear that 
adjudicators perceive the authors of the high-impact style 
cases to 

® possess better judgment 

@® be more organized, 


@® be more precise, 


@® be more careful, 
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® be more clear thinking, 
® and be more efficient, 
than the writers of low-impact style cases. These perceived 
attributes may increase the field agents’s credibility in the 
eyes of the adjudicators and thus produce greater confidence 
in the ROI case information. This increased credibility and 
confidence in the case information would reduce the requests 
for additional information concerning the case subject. 
Taking that argument one step further, we could say that 
credibility improves the efficient flow of information in the 
adjudication process and will thus decrease case processing 
time. 
As will be seen in the following sections, these 
differences in perception are not necessarily true when the 
Czarnek and Rokitka cases or the adjudications sites are 


analyzed separately. 


C. ANALYSIS OF THE HIGH AND LOW-IMPACT CZARNEK CASES 

The analysis results of questionnaire Part III using just 
the high and low-impact Czarnek cases are presented in Table 
B-2. The analysis of the fifteen ratings of the bipolar 
adjectives showed that none of the results were found to be 
Statistically significant within either the 5% or 10% 
confidence intervals. 

These result tells us that there was no difference in 


adjudicators’ perception of the field agents who wrote either 
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TABLE 5-2 











SUMMARY OF QUESTIONNAIRE PART III RESULTS 
ANALYSIS OF THE CZARNEK CASE 
ADJUDICATOR’S PERCEPTION OF FIELD AGENT WRITING ROI 


QUEST Pace Higsoal. ee Lbe.S abe F-STAT SIG. 
ION # a: F 


ito | 0.93 | | 25035 | 0.846m | lolseae 












0. 863 
3 hase | oe er 
ps6 | 2a | a2 | 2.35 | 0.82 | 1.207 | 0.264 
|e | 26 | 2.22 | 2.25 | 0.85 | 0.706 | 0.407 
poo | 2 | ze | 2.2 | 0.95 | 0.172 | 0.602 _ 


the high or low-impact Czarnek cases. We can also infer that 















these results are not the driving force behind the 
statistically significant results to the bipolar ratings when 
both the Czarnek and Rokitka Cases were analyzed together. 
Similar to the Questionnaire Part II results, this lack of 
Significant difference is unexpected because Czarnek case 


content is much more complicated and longer than Rokitka case 
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content. One would assume that writers of high-impact 
documents containing longer and more complicated information 
would be more positively perceived than writer of low-impact 


ones. 


D. ANALYSIS OF THE HIGH AND LOW-IMPACT ROKITKA CASES 

The result of Part III using just the high and low-impact 
Rokitka case date is presented in Table 5-3. Only the 
bipolar rating for clarity of thinking (#8) was found to be 
statistically significant within the 5% confidence interval. 
Three other bipolar ratings of organization (#5), efficiency 
(#10), and judgment (#14) were significant within the 10% 
interval. 

Comparing these semantic differential results with those 
from our analysis of both case, we see consistent findings. 
The statistically significant responses described above were 
also found significant within either the 5% or 10% confidence 
interval in the combined Czarnek and Rokitka analysis (Table 
m1) . 

Unlike the Czarnek cases, we can say that the Rokitka case 
responses are the driving force behind the statistically 


Significant results we found when both cases were combined. 
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TABLE 5-3 


SUMMARY OF QUESTIONNAIRE PART III RESULTS 
ANALYSIS OF THE ROKITKA CASE 
ADJUDICATOR’S PERCEPTION OF FIELD AGENT WRITING ROI 


Torr Ca MEAN Ola 
bene Sa 
: Ge 7 > Za dC! Lao O = aes 
ee 
2.05 | 2.24 | 2.5 | 2.24 | 2.998 | 0.092 
pies | 0.93 | aia [aan aces aaa 


2.352 | 0.134. 


poa.7 {oe | 22 | 2.25 | 2.089 | 0.303_ 
_a.6 | 0.75 | 2.2 | 2.06 | 3.229 | 0.082 
Za 





E. ANALYSIS OF THE SEPARATE ADJUDICATION SITES USING BOTH THE 
HIGH AND LOW-IMPACT CZARNEK AND ROKITKA CASES 
This section presents and analyzes the Part III results 
for each separate adjudication site. This section will be 


presented in three subsections: Army, Navy and DISCR results. 
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1. Presentation and Analysis of the Army Adjudication 
Site Results 

Unlike the last chapter, the Army’s questionnaire Part 

III results did not dominate the other sites in statistically 

Significant results. Table 5-4 shows that the result to the 

judgement bipolar rating (#14) was the only result significant 

to within either the 5% or 10% confidence interval. This 


rating was is consistent with the combined case analysis. 


TABLE 5-4 







SUMMARY OF QUESTIONNAIRE PART III RESULTS 
ANALYSIS OF ARMY SITE (n = 12) USING BOTH CASES 
ADJUDICATOR’ 5 PERCEPTION OE FIELD AGENT WRITING ROI 


ION # MEAN MEAN of F 
a ee 
OR"s> Aisi ore? Leet ie Ors'saZ 
| Gee an Pn me ee 
pa | 2.25 | 0.75 | 2.33 | 0.98 | 0.052 | 0.823 
| 5 || 2.ce | 0.79 | 2.58 | 0.9 | 2.093 | 0.163 
ps | 227 | 2.03 | 2.5 | 3.27 | 0.986 | 0.333 
| 7 | 227 | 0.94 | 2.55 | 0.93 | 0.86 | 0.365 _ 
ms Jee 2 | wo.ss | aiaz | a | $2.25 | | 0.15 | 
po | 2.67 | 2.56 | 2.42 | 2.26 | 0.008 | 0.020 
| a2 |} r03 | 0.94 | 2.25 | 0.97 | 2.05 | 0.268 _ 
| a2 | a.92 | o.9 | 2.36 | 0.92 | 1.66 | 0.213 
| aa | ae3 | 0.98 | 2-42 | 2 | 3.4 | 0.082 
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2. Presentation and Analysis of the Navy Adjudication 
Site Results 

The result of the Navy site analysis are presented in 

Table 5-5. Only the "precision" bipolar rating (#6) was found 

to be statistically significant within the 5% confidence 

interval. Three other bipolar ratings for clarity of thinking 

(#8), awareness (#9), and efficiency (#10) were within the 


10% interval. Comparing these results with those from the 


TABLE 5-5 










SUMMARY OF QUESTIONNAIRE PART III RESULTS 
ANALYSIS OF NAVY SITE (n = 12) USING BOTH CASES 
ADJUDICATOR" S PERCEPTION OF FIELD AGENT WRITING ROI 


ION rae ae 5 P 
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analysis of both case, we see consistent findings, with only 
one exception. Question 6 was statistically Significant 
within the 10% interval in the previous analysis, but the 
Navy results show it falling within the 5% interval. 
3. Presentation and Analysis of the DISCR Adjudication 
Site Results 
The DISCR results are presented in Table 5-6. This 


Site produced no statistically significant results. Oddly 


TABLE 5-6 










SUMMARY OF QUESTIONNAIRE PART III RESULTS 
PuauyYsilS OF DISCR SITE (n = 12) USING BOTH CASES 
ADJUDICATOR’S PERCEPTION OF FIELD AGENT WRITING ROI 
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enough, none of the results of the analysis were even close to 
falling within the statistically significant range indicating 
there is relatively little difference in DISCR adjudicators’ 
perception of the field agents who wrote either the high or 
low-impact styles cases. The reason why the adjudicator’s show 
no preference between the two case may lie in unique 


organizational norms at the DISCR. 


F. PRESENTATION AND ANALYSIS OF THE MALE ADJUDICATOR RESULTS 

Just as in the previous chapter, the analysis of the 
several demographic breakdowns produced no significant results 
except for the male adjudication population. These result are 
presented in Table 5-7. The bipolar ratings for organization 
(#5), clarity of thinking (#8), and efficiency (#10) were 
found to be statistically significant at the 5% level, while 
the rating for thoroughness (#1), precision (#6), care (#7), 
confidence (#11), and judgement (#14) were statistically 
Significant at the 10% level. 

For the most part, these results are consistent with those 
found in the analysis of both cases using the entire 
population (Table 5-1), with three notable exceptions. There 
are two additional bipolar ratings (#1 "thoroughness" and #11 
"confidence") with statistically significant results, and one 
question (#5 "clarity of thinking") with results of greater 


statistical significance (i.e. 5% vice 10%). Similar to the 
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TABLE 5-7 











SUMMARY OF QUESTIONNAIRE PART II RESULTS 
ANALYSIS OF MALE ADJUDICATORS (n = 9) USING BOTH CASES 
eee S IMPRESSIONS OF CASE INFORMATION 
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male adjudicator’s result to Part II, these results are 







remarkable considering the small sample size (n=9). 


G. CHAPTER CONCLUSION 

The result to the combined Czarnek and Rokitka case 
analysis point out that there is statistically significant 
evidence to show that adjudicators have different perceptions 


of the field agents who wrote high-impact style cases versus 
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those who composed low-impact one. Specifically, they feel 
that the agents writing ina high-impact style possess better 
judgment, are better organized, and are more precise, careful, 
clear thinking, and efficient than writers of the low-impact 
cases. These perceived attributes increase field agent 
credibility, which could improve the efficiency of the entire 
adjudication process. 

When the cases were analyzed independently, we found that 
there was 1) no significant differences in the adjudicator’s 
perception of field agents who wrote the high or low-impact 
Czarnek cases; and 2) statistically significant differences in 
the perceptions of the high and low-impact Rokitka writers. 
These differences were found in the areas of organization, 
clarity of thinking, efficiency, and judgement. 

When the analysis was broken down by adjudication site, 
the statistical significance was not nearly as great as the 
results from the entire population. This result could be 
attributed to the different sample sizes used in the analysis 
of both cases and each site (i.e. 40 versus 12 adjudicators). 
The greatest statistical differences was found in the results 
from the Navy adjudicators, with the Army coming in a distant 
second. DISCR came in third with no statistically significant 
m@ecUUL css: 

Similar to the result in the previous chapter, the most 
Surprising result was found from the analysis of the 


demographic breakdown of the adjudicators. The survey 
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responses from the male adjudicators produced statistically 


Significant results. 
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VI. PRESENTATION AND ANALYSIS OF PART IV RESULTS: THE 
WRITING STYLE PREFERENCES OF ADJUDICATORS 
This chapter presents the results from Questionnaire Part 

IV (see APPENDIX (C). Part IV tested the adjudicators’ 
impressions or feeling about the high and low-impact writing 
styles by having them choose which case treatment they 
prefered in relation to eight factors. These preference 
factors include: 

® presentation of information; 


@® ease of reading, remembering, and applying § the 
adjudication criteria; 


@® amount of rereading; 

® confidence in adjudication decision; 

® competence of the field agent writing the case; 
® and overall writing style. 

The chapter is organized in five sections. The first 
subsection gives direction on how to read the tables of 
results and the last draws conclusions. The remaining middle 
subsections are organized as follows: 


® Both Cases: Analysis of both Czarnek and Rokitka high/low 
impact cases uSing entire adjudicator population. 


® Separate Adjudication sites: Analysis of each Adjudication 
site (Army, Navy, and DISCR) using both cases. 


® Male Adjudicators: Analysis of both cases using Male 
Adjudicators. 
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A. DIRECTION FOR READING TABLES OF RESULTS 

There are five tables of results in this chapter. Each 
table has a title block and seven columns of information. 
Below is an extract of the title block and column heading 


with an explanation of each. 


SUMMARY OF QUESTIONNAIRE PART IV RESULTS 
CASES AND ADJUDICATORS TESTED (n = sample size) 
ADJUDICATOR’'S PREFERENCE FOR HIGH OR LOW-IMPACT ROI 


PREFER HI PREFER LI CHa SOR. 


) @ | 





The title block is the same for all tables with the 
exception of the second line, which is highlighted above. 
This line states which case and adjudicators were included in 
the analysis. An explanation of the seven column headings 


are as Eollows: 


@® QUEST #: The Part IV question analyzed, excluding question 
6 and 8. 


@® PREFER HI: The number and percentage of adjudicators who 
prefer the high-impact style cases relative to the tested 
preference factor. 


@® PREFER LI: The number and percentage of adjudicators who 
prefer the low-impact style cases relative to the tested 
preference factor. 


@® CHI-SOR. STAT.: The value of the statistic resulting from 
the Chi-Squared testing. 


@® SIG. of CHI-SOR. STAT.: The significance of the chi-square 


statistic, which is the probability that the chi-square 
value is due to a random occurrence. 


15 


ve) 


The results of questions which fall within the 5% 
confidence interval are shaded while, results that meet the 
10% interval are highlighted in bold type. 

Questions 6 and 8 were posed in a yes/no format and could 
not be statistically analyzed using Chi-squares. The answer 


to these questions are included in the discussion of results. 


B. THE ANALYSIS OF BOTH THE CZARNEK AND ROKITKA CASES RESULTS 
The third primary research question, outlined in the first 
chapter, asks 
@® Will there be statistically significant differences in 
adjudicators’ preferences for the case information 


presented in a high-impact versus the low-impact style? 


The answer to that question is given in Table 6-1 below: 


TABLE 6-1 


SUMMARY OF QUESTIONNAIRE PART IV RESULTS 
ANALYSIS OF BOTH CZARNEK & ROKITKA CASES (n = 40) 
ADJUDICATOR’S PREFERENCE FOR HIGH OR LOW-IMPACT ROI 


QUEST 
#f 
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Each of the significant results in the table, as well as 
the responses to yes/no questions, will be compared with 
either Part II or III Survey results. Unless otherwise noted, 
only Part II or III results significant at the 5% or 10% 
confidence levels are used in the comparison. 

1. Question 1: Presentation of Case Information 

This question asks adjudicators which case treatment 
writing style they prefered. The results, statistically 
Significant at the 1% level, show that by a 29 to 11 margin 
adjudicators prefer the high-impact style to the low-impact 
one. 

These results confirm the findings from Part II of the 
survey. Specifically, comparing Part II high-impact and low- 
impact results of adjudicators’ impressions of the way 
derogatory information is presented reveals that the high- 
impact style seems to 

® make it easier to apply the adjudication criteria, 


® increase the adjudicators’ confidence in the quality of 
their adjudication decision, 


® improve field agent awareness of adjudicators’ daily 
reading requirements, 


@® and makes it easier to find and remember derogatory 
information. 


2. Question 2: Ease of Reading 
This question asked the adjudicators which case style 
was eaSier to read. The results, statistically significant at 


the 5% level, show that by a 27 to 13 margin adjudicators 
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found it easier to read cases in the high-impact rather than 
the low-impact style. 

These results directly verify findings of one Part II 
question and indirectly corroborate the findings from two 
others. The directly applicable questions asked the 
adjudicators how easy it was to read the derogatory 
information, while the indirect questions determined to what 
extent did reading the cases caused fatigue and eye strain. 
The high-impact style made it easier to read derogatory 
information and reduced fatigue and eye strain. 

3. Question 3: Ease of Remembering Case Information 

This question asked adjudicators if information was 
easier to remember with the case treatments using either the 
high or low-impact style. The results, statistically 
Significant at the 5% or 10% level, show that by a 27 to 13 
margin adjudicators more easily remembered case information 
written in the high-impact style than the low-impact one. 

These results confirms the finding from one Part II 
question. This question found that, given the way information 
was presented in the high and low-impact cases, adjudicators 
felt that it was easier to remember the case information in 
the high-impact style than the low-impact one. 

4. Question 4: Frequency of Rereading Case Information 

This question asked adjudicators which case style had 


to be read more often. The results, statistically significant 
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Pea cicm@ eee level, show that by#aws0 tee9 margin adjudicators 
had to reread the case information more often in the low- 
impact than the high-impact cases. 

These results corroborate the finding from one Part II 
question. This question determined that adjudicators’ felt 
they had to reread sentences more often in the low-impact 
style than the high-impact one. 

5. Question 5: Ease of Applying the Adjudication Criteria 

This question asked adjudicators which writing style 
made applying the adjudication criteria easier. The results, 
Statistically significant at the 5% level, show that by a 27 
to 12 margin adjudicators find applying the adjudication 
criteria easier in the high-impact style than the low one. 

These results corroborate the finding from _ one 
question in Part II. The results to that question indicate 
that the adjudicators’ impressions of the way derogatory 
information is presented in the high-impact style seems to 
make it easier to apply the adjudication criteria than in the 
low-impact cases. 

6. Question 6: Confidence in Adjudication Decision 

This yes/no question directly asked adjudicators if 
the different writing styles affected their confidence in 
their adjudication decision. By a 31 to 9 margin the 
adjudicators felt the different styles did not affect their 


GContidence. 


a9 


Much care must be used when comparing these results 
with the findings from related Part II questions. The key 
obstacle to effective comparison lies in the fact that this 
Part IV question directly tested Adjudicators’ confidence in 
their adjudication decision, while the Part II questions 
determined only indirectly their degree of confidence. Part 
II determined adjudicators’ relative confidence in 

® the quality of the field agent’s investigation, 

® and the soundness of their adjudication decision. 
These results indirectly indicate that the high-impact style 
increased adjudicators’ confidence in their decision compared 
to the low-impact style, but the position of the mean scores 
show that they felt relatively confident using either style. 
Combining the Part II and IV results, we see that indirect 
testing shows that the high-impact style affects adjudicators’ 
confidence in both the field agents’ investigation and the 
soundness of their adjudication decision, though direct 
testing does not. 

However, we must not ignore that 24% of the 
adjudicators did indicate that the different case writing 
styles did affect the confidence in their adjudication 
decision. The other 76% of adjudicators may be reluctant to 
say directly that any case style would affect their confidence 
because this would reflect poorly on their adjudicative 


abiJaiey . 
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7. Question 8 and 9: Perception of the Field Agent’s 

Competence 

Question #8 was a yes/no question that asked 
adjudicators if the different writing styles affected their 
perception of the field agents’ competence. Question #9 asked 
Only those who responded "yes" if the writer of the high- 
impact or the low-impact style was more competent. 

Question #8 results show that one-half of the 
adjudicators (20) indicated writing style affected their 
perception of field agent competence. The fact that half of 
the adjudicators said that style affected their perception is 
fairly startling since it indicates that style alone had a 
major impact on perception of competence. 

The competence of an individual is, however, 
ambiguous, and depends on the definition used. It may be a 
combination of all, or part of the descriptive adjectives used 
in Part III of the survey. This study interprets competence 
as the combination of the descriptive adjectives from part III 
that tested in the significant level. 

With this in mind, a comparison of the Part IV and III 
results reveals a pattern that supports the finding that the 
high-impact style affects adjudicators’ perception of field 
agents’ competence. Specifically, Part III shows that 
adjudicators perceived field agents who wrote the high-impact 
cases to be more organized, precise, careful, clear thinking, 


and efficient than the writers of the low-impact cases. They 
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also felt that high-impact case writers possessed better 
judgement. The combination of these statistically significant 
resultS points out that writing style does affect 
adjudicators’ perception of field agents’ competence. 

Futhermore, the results to Question #9 show that by a 
margin of 17 to 4 the adjudicators saying "yes" to Question #8 
feel that the writers of the high-impact cases are more 
competent than the writer that use the low-impact style. This 
result also indicates that the portion of the adjudicator 
responding "yes" feel that high-impact style reflects a more 
competent writer than the currently used low-impact style. 

8. Question 10: Writing Style Preference 

This question asked adjudicators which style would 
they prefer all ROI’s be written. The results, statistically 
Significant at the 5% level, show that by a margin of 27 to 12 
adjudicators prefer the high-impact style cases over the low- 
impact style ones. 

This result verifies the findings of both 
Questionnaire Part II and III. Those survey results show that 
there is a statistically significant difference between the 
high and low-impact style cases in the adjudicators’ 
impressions of case information and their perceptions of field 
agents writing the cases. Analyzed together, the result of 
Part II, III and IV clearly point out that the high-impact 


Style is superior to the status quo low-impact style. 
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C. ANALYSIS OF THE SEPARATE ADJUDICATION SITE RESULTS 

This section presents and analyzes the Part IV results for 
each separate adjudication site. This subsection will 
present qualitative discussion of the results from the three 
Sites: Army, Navy and DISCR. 

1. Presentation and Analysis of the Army Adjudication 

Site Results 

The Army Site showed the least number of statistically 
Significant results to Questionnaire Part IV. Those results 
are presented in Table 6-2. Question #9, which tested the 
positive respondents to question #8 (the Competence of field 
agents), was Significant at the 5% level, while Question #3, 
which determined which case writing style produced easier to 
remember information, was Significant at the 10% level. 
Although the remainder of the results are not significant, 
they all favor the high-impact style documents. Question #6 
results indicate (4) yes and (8) no responses, while question 
#8 shows (5) yes and (7) no responses. 

The only major significant result showed by a 9 to 3 
margin that adjudicators felt information presented in the 
high-impact style was easier to remember than that presented 
in the low-impact style. Also, the 5 yes respondents to 
question #8 believe that the high-impact writers were more 


competent than the low-impact ones. 
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TABLE 6-2 


SUMMARY OF QUESTIONNAIRE PART IV RESULTS 
ANALYSIS OF ARMY SITE (n = 12) USING BOTH CASES 
ADJUDICATOR’S PREFERENCE FOR HIGH OR LOW-IMPACT ROI 


+; QUEST PREFER HI PREFER LI CHI-SOR. 
# | STAT 
LS 


42% II 





2. Presentation and Analysis of the Navy Adjudication 

Site Results 

The Navy Site showed the most’~ statistically 
Significant responses to Questionnaire Part IV. Those 
results are presented in Table 6-3. They show that three 
questions (#’s 1, 2, and 10) were significant at the 5% 
confidence level, and question #2 was significant at the 10% 
level. The results also indicate that question #6 had only 
negative responses, while question #8 had (4) positive and (8) 
negative responses. 

The major significant result showed that’ the 
adjudicators 


® preferred the presentation in the high-impact style cases 
(#1), 
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@® felt that the information presented in the high-impact 
style was easier to read (#2), 


@® had to reread information more often in the low-impact 
Style cases (#4), 


@® and preferred that all cases be written in the high-impact 
style (#10). 


Although the remainder cof the result are not Significant, they 
indicate that the adjudicators prefer the high-impact 


documents. 


TABLE 6-3 


SUMMARY OF QUESTIONNAIRE PART IV RESULTS 


ANALYSIS OF NAVY SITE (n = 12) USING BOTH CASES 
EEE eS PREFERENCE FOR HIGH OR LOW-IMPACT ROI 


ae” PREFER HI PREFER LI CHE SOR. 
STAT. 









3. Presentation and Analysis of the DISCR Adjudication 
Site Results 

The DISCR Site showed the second most statistically 

Significant results to Questionnaire Part IV. Those results 


are presented in Table 6-4. Although none of the responses 
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TABLE 6-4 


SUMMARY OF QUESTIONNAIRE PART IV RESULTS 
ANALYSIS OF DISCR SITE (n = 12) USING BOTH CASES 





were Significant within the 5% level, and five questions (#’s 
1, 2, 5, 9, and 10 ) were significant within 10% level. The 
results also indicate that question #6 had (4) positive and 
(8) responses, while question #8 had (9) positive and (3) 
negative responses. The statistically significant results 
showed that adjudicators 


® preferred the presentation in the high-impact style cases 
(#1) 


® felt that the information presented in the high-impact 
style was easier to read (#2) 


® found it easier to apply the adjudication criteria to 
information in the high-impact style cases (#5) 


® and preferred that all cases be written in the high-impact 
style (#10). 
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Also the respondents who answered "yes" to question #8 
indicated in question #9 that they believe the high-impact 
writers were more competent than the low-impact ones. 
Although the remainder of the results are not statistically 
Significant, they indicate that adjudicators prefer the high- 


impact documents. 


D. ANALYSIS OF THE MALE ADJUDICATOR RESULTS 

Just as in the previous two chapters, the analysis of the 
several demographic breakdowns produced no significant results 
except for the male adjudicator population. Those results are 
presented in Table 6-5. Seven preferences questions (#’s 1, 
2, 3, 4, 5, 9, and 10) were significant at the 5% level, and 
none were significant within 10% level. The results also 
indicate that question #6 had (1) positive and (8) negative 
responses, while question #8 had (5) positive and (4) negative 
responses. The major significant result showed that the male 
adjudicators 


® preferred the presentation in the high-impact style cases 
(#1), 


® felt that the information presented in the high-impact 
style was easier to read and remember (#’sS 2 and 3), 


@® had to reread information more often in the low-impact 
style cases (#4), 


® found it easier to apply the adjudication criteria to 
information in the high-impact style cases (#5), 


® and preferred that all cases be written in the high-impact 
style (#10). 
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TABLE 6-5 


SUMMARY OF QUESTIONNAIRE PART IV RESULTS 


ANALYSIS OF MALE ADJUDICATORS (n = 9) USING BOTH CASES 
ADJUDICATOR’S PREFERENCE FOR HIGH OR LOW-IMPACT ROTI 


QUEST PREFER HI PREFER LI Chie SOR. 
A STAT . 
ae ee ee 














Also the respondents who answered "yes" to question #8 
indicated in the question #9 results that they believe that 
the high-impact writers were more competent than the low- 
impact ones. 

These results confirms the findings in Questionnaire Part 
II and III. The male adjudicators are the mavericks in the 
adjudication population. They seem to believe more strongly 
than their female counterparts that the high-impact style 
cases are a great improvement over the current low-impact 


ones. 
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E. CHAPTER CONCLUSION 

The overall results to Questionnaire Part IV point out 
that there is statistically significant evidence to show that 
adjudicators prefer cases be written in the high-impact rather 
than the low-impact style. Specifically, the analysis 
indicates the adjudicators prefer the high-impact over the 
low-impact style relative to the following factors: 

® presentation of information; 


@® ease of reading, remembering, and applying the 
adjudication criteria; 


® amount of rereading; 
® and overall writing style. 

When the Questionnaire Part IV analysis was broken down by 
adjudication site, the statistical significance was not nearly 
aS great as the results from the entire population. This 
result could be attributed to the different sample sizes used 
in the analysis of both cases and each site (i.e. 40 versus 12 
adjudicators). The greatest statistical differences was found 
in the results from the Navy adjudicators, with DISCR coming 
second, and Army third. 

Similar to the result in the previous two chapters, the 
most Surprising result was found from the analysis of the 
demographic breakdown of the adjudicators. The survey 
responses from the male adjudicators produced statistically 


Significant results. 


89 


WEES CONCLUSIONS AND RECOMMENDATION 


A. CONCLUSIONS 

Examining Questionnaire Part II, III and IV results from 
the high and low-impact treatment of both the Czarnek and 
Rokitka cases point out that the adjudicators prefer the high- 
impact style over the currently used low-impact one. The data 
indicates the adjudicators prefer the high-impact style for 
its 

® presentation of information; 


@® ease of reading, remembering, and applying the 
adjudication criteria; 


® limited amount of rereading; 
® and overall writing style. 

There was, however, no statistical evidence to show that 
the high-impact case decreases adjudication time. 

In the area of the adjudicators’ impression of the case 
information, the analysis does produce - = statistically 
Significant evidence to indicate that adjudicators feel that 
the high-impact style is an improvement over the low-impact 
style. Specifically, they believe that the high-impact style 

® provides better coverage of issues; 


® makes it easier to apply adjudication criteria to the case 
information; 
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® makes the field agent who wrote the case seem more aware 
of 1) how to make the case easier to read; 2) their 
informational needs; 3) the extensive amount of their 
daily reading; 

® makes the field agent seem like a better writer; 


® makes the derogatory information in the narrative report 
easier to find, read, remember and adjudicate; 


® makes following the chronological sequence of events in 
the case easier to follow; 


® and causes less fatigue while reading. 

To a lesser degree they believe that the high-impact style 
makes them more confident in the soundness and quality of 
their adjudication decision, causes them to reread sentences 
less often, and produces less eye stain. 

The analysis of the adjudicators’ perception of the field 
agents provided statistically significant evidence to show 
that the field agents who wrote high-impact style cases are 
perceived differently than those who composed low-impact ones. 
Specifically, they feel that the agents writing in a high- 
impact style possess better judgment, are better organized, 
and are more precise, careful, clear thinking, and efficient 
than writers of low-impact cases. These perceived attributes 
may increase field agent credibility, which could improve the 
efficiency of the entire adjudication process. 

Both Part II and III results indicate that there were 
greater statistically significant differences between high and 
low-impact case treatments of the Rokitka cases than the 


Czarnek cases. This result 1s contrary to expected results 


oa. 


because the Czarnek case content is more complicated and 
longer than Rokitka case content. One would expect that the 
longer and more complicated a document’s content, the more 
helpful the high-impact style would be in reading, finding, 
and processing information. 

When the analysis was divided by adjudication site, the 
statistical significance was not nearly as great as the result 
from the entire population. However, this could be attributed 
to the smaller sample sizes used. Overall, the greatest 
number of statistical differences were found from the analysis 
of the Navy adjudication site data. The Army adjudication 
Site data produced the second most statistically significant 
result, while DISCR data came ina distant third. 

The most surprising result was found when analyzing the 
demographic breakdown of adjudicators. The male adjudicators 
found that the high-impact style decreased the time required 
to adjudicate a case. They also felt that the high-impact 
cases were a much greater improvement over the current low- 
impact cases. Finally, their perception of the field agents 
who wrote the high-impact cases was extremely different than 
the low impact authors. These results may be due to the fact 
that males are a minority and could be considered outsiders. 
Outsiders may be more willing to change the status quo of an 


organization than insiders. 
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B. RECOMMENDATIONS 

The overall results to Questionnaire Parts II, III, and IV 
clearly point out that the adjudicators perceive the high- 
impact style cases to be superior to the current low-impact 
style. Specifically, most adjudicators prefer that all ROI’s 
be written in a high-impact style. 

With this in mind, DIS should heed their customers’ 
desires and train their field agent to write ROI’s in a high- 
impact style. Although the high-impact style does not at a 
statistically significant level improve the decision making 
ability nor decrease case assessment time, we still believe 
that DIS should adopt the high-impact style in the spirit of 
improved customer support. 

If implemented, the high-impact ROI’s could increase the 
efficiency of the adjudication process. The adjudicators will 
be seeing cases written in a way they perceive as more 
efficient. This perceived efficiency may increase the field 
agents’s credibility in the eyes of adjudicators and thus 
produce greater adjudicator confidence in the ROI case 
information. This increased credibility and confidence in the 
case information could reduce requests for additional 
information concerning the case subject. This reduction in 
requests for additional information will thus decrease overall 


case processing time. 
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APPENDIX A 


PART Il 


INSTRUCTIONS 


This form is designed to capture your impressions of the case information you've read 
and the adjudication process you've undergone. Please read each of the following 
statements carefully, and then place an "X" next to the place on the scale that best 
describes your response to each statement. 


For example, if you were responding to the statement 


It was for me to find important information in the case. 
1 2 3 4 S 
Easy Difficult 


and you felt that to find information it was 


easy: check # 1 
somewhat easy: check # 2 
neither easy 

nor difficult: check # 3 
somewhat difficult: check # 4 
difficult: check # 5 

STATEMENTS 
]. This case provided ss COerrage of issues to enable me to make a high quality 


adjudication decision. 
||] LJ Lica 
1 2 3 4 5 


Good Poor 
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to 


The way field agents presented derogatory information in the case made it 


to apply adjudication critena to that information. 
1 2 3 4 5 


Easy Difficult 


The way derogatory information was presented made me 
the field agents’ investigation. 


ieee tL) LI Lo 


Confident Unconfident 


in the quality of 


After reading the case, I am in the soundness of my adjudication decision. 


Eee Lee TT gd 


Confident Unconfident 


The field agents seemed to be of the information I needed to make a high 


quality adjudication decision. 
fF td) wl CU Ss 
1 2 3 4 Ss 


Aware Unaware 


The field agents who wrote this case seemed to be 
case easy for me to read. 


YR ee ee 


Aware Unaware 


a5 


of how to make the 


7. Based on the way feld agents presented information in the case, they seemed to be _ 
of the large amounts of information | have to read daily. 


LL ee 


Aware Unaware 


8. I believe the feld agents who wrote the narrative reports for this case are 


writers. 
1 2 3 4 5 
Good Poor 
os) that the field agent who wrote the case’s subject interview section should 


receive training to make his/her writing more readable. 
1 2 3 A 5 
Agree Disagree 


10. I was able to read this case than similar cases. 
CO eae 
1 2 3 4 5 


Faster Slower 
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Vz 


Jy 


14. 


LS: 


. The way field agents presented information in their narrative reports made it 


to tind derogatory information. 
ea hh 
1 2 3 4 ) 


Easy Difficult 


Derogatory information wnitten by the field agents was to read.. 
1 2 3 4 S 


Easy Difficult 


] thought the subject interview section of the case was to read. 
1 2 3 4 ) 


Easy Difficult 


The way the field agent presented derogatory information made it 
remember that information. 


2 2 eee 
1 2 3 4 ) 
Easy Difficult 
I found the case to be to adjudicate. 
S _ eS, 2 
1 7 3 4 ) 


Easy Difficult 


o7 


16. 


de 


18. 


19. 


AB} 


It was for me to locate information essential to my efficient adjudication 
of the case. 


ee OE as ee a 


Easy Difficult 


To understand what the field agent was trying to convey, I had to reread 
a section or sections of the case. 


L | 


Often Never 





I had to reread sentences to understand the exact idea field agents were 
trying to convey. 


Se ee ae 


Often Never 





I found it to follow the chronologica] sequence of events in the case. 
1 2 3 4 5 


Easy Difficult 


To help me understand the Jogical connection between ideas in the case, | 
_ had to either underline parts of the case or make notes. 


Ph) Li 


Often Never 
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ae | encountered words that were difficult to understand while reading the 
field agents’ narratives. 


i a rg oan gale I 


Often Never 


22. Reading this case tured me out. 
a ie ee Ld 
1 2 3 
Agree Disagree 
23. Reading this case strained my eyes. 
[=r oe 
1 a 3 4 Ss 
Agree Disagree 
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APPENDIX B 


PART Iil 


INSTRUCTIONS 


Please indicate your perceptions or feelings toward the field agents who wrote the ROI 
you've just read by placing an "X" near the item that best descnbes that perception. 


For example, if after reading the ROI, you fee] the field agents are 


Thorough: check # 1 


Somewhat thorough: check # 2 


Neither thorough 
nor careless: check # 3 


Somewhat careless: check # 4 
Careless: check # § 
Use this guideline to respond to the following items. 





PERCEPTIONS OF FIELD AGENTS 


1 A 3 4 5 
Thorough Careless 


1 2 3 4 5 
Competent Incompetent 


1 Z 3 4 5 
Objective Biased 
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PERCEPTIONS OF FIELD AGENTS (Continued) 


1 7 3 4 ) 
Perceptive Unperceptive 


1 Fe 3 4 ) 
Organized Disorganized 


1 2 3 4 Ss 
Precise Imprecise 


1 2 3 4 5 
Careful Careless 


1 2 3 4 S 
Clear thinking Fuzzy thinking 


1 2 3 4 5 
Observant Unaware 


con 


PERCEPTIONS OF FIELD AGENTS (Continued) 


Efficient Inefficient 


1 2 a 4 5 
Confident Unconfident 


Reliable Unreliable 


1 2 3 4 S 
Experienced Inexpenienced 


1 2 3 4 ) 
Sound judgment Poor judgment 
Je | | | | | | 
1 2 3 4 3 
Veteran Rooke 
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APPENDIX C 
PART IV 


Information presented in both cases was wnitten and formatted in different styles. Please 
provide us with your impressions or feelings about these different written styles by checking 
the appropnate responses to the statements Listed below. 


1. I preferred the way information was presented in 


Case | 


Case II 


2. I felt the case that was easier to read was 


Case | 


Case I 


3. Information was easier to remember in 
Case | 


Case II 


4. I had to reread information more often in 


Case | 


Case I 


5. I found it easier to apply adjudication criteria to information in 
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I felt the two cases’ different wniting styles affected my confidence in my adjudication 
decision. 


If you answered “yes” to question 6, did you fee! more confident of your decision when 
you adjudicated Case I or Case II? 


Case | 


Case II 


I felt the cases’ different written communication styles affected my perception of the 
competence of the field agents. 


If you answered "yes" to question 8, did you fee! that the field agents who wrote Case 
I or Case II] were more competent? 


Case | 


Case I] 


10. I would prefer that all ROIs be wntten in the style of 
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